Magnum One

home *** CD-ROM | disk | FTP | other *** search

/ Magnum One / Magnum One (Mid-American Digital) (Disc Manufacturing).iso / d20 / gmd_300.arc / GMD.DOC < prev next >

Wrap

Text File | 1991-09-11 | 29KB | 637 lines

**************************************************************************** **************************************************************************** **** **** **** Note: The "delete" command is disabled in this release. **** **** **** **** This was done to limit the possible damage from **** **** incorrectly setting up GMD, to allow users and **** **** software authors more time to fix the problems **** **** which GMD has found, and in the hope that the FTSC **** **** will soon revise FTS-0004 so that there will be a **** **** more definitive standard for echo mail. **** **** **** **** The "delete" command will be functional in version **** **** 3.10, which we expect to release in about 1 month. **** **** **** **** Note: The OS/2 version didn't quite make it in time for **** **** this release. We expect it to be ready for the **** **** version 3.10 release. **** **** **** **************************************************************************** **************************************************************************** Grunged Message Detect (GMD) Utility ====================================== Version 3.00 ============== September 12, 1991 ==================== Table of Contents =================== 1.0 Warning! 2.0 General Description 3.0 Reasons to Run GMD 4.0 Definition of a "Grunged" Message 5.0 Operation 5.1 GMD 5.2 GMD-Sort 5.3 GMD-Rpt 6.0 GMD Configuration File 6.1 General Operation Section 6.2 Set Definitions Section 6.3 Default Message Standards Section 6.4 Exception Areas Sections 7.0 Runtime Considerations 7.1 DOS 7.2 OS/2 7.3 Other Operating Systems 8.0 Licensing Restrictions 9.0 Credits 10.0 History of Changes 1.0 Warning! ------------- GMD is a very powerful program. Used properly, GMD can detect many types of grunged messages and some types of duplicate messages. It can also be used as a management tool by Moderators, an independent verification tool by software authors, and a testing tool by the average SysOp. Used improperly, GMD is capable of deleting valid echo mail, generating false warning messages, and violating Policy in general. Thus you should be very careful in setting it up and in using it. If you haven't read and understood FTS-0001 and FTS-0004 then you should not attempt to configure or use GMD. To protect you, me and the rest of the Net, GMD will not delete an echo mail message without sending an information message to the originator. Until you are sure you have your configuration correct, you should use GMD in the test mode by using the "-t" command line switch. 2.0 General Description ------------------------ GMD's main purpose is to detect grunged messages. In the process of doing this, GMD is also able to identify certain types of duplicate messages and non-standard messages. GMD operates directly on incoming packets, as described in FTS-0001 and FTS-0004. GMD is very configurable. Indeed, it depends entirely on a configuration file to tell it what parameters to check the messages for, and what to do when it finds a "bad" message. 3.0 Reasons to Run GMD ----------------------- Grunged messages can cause problems for some mail processing programs. For example, they can cause certain message renumbering programs to lockup. Since message renumbering is usually a part of nightly processing, your system is exposed to locking up while you are not around. When you discover the lockup, you are faced with the task of finding the grunged message, deleting it and then completing the night's processing. It is not uncommon for this recovery process to last several hours at an inopportune time. Deleting grunged messages and duplicate messages spares your users from seeing them and protects other boards that you feed echo mail. Informing the originator of such messages allows him to correct the problem at the source. GMD is most effective when used close to the source of the problem. Thus we encourage using GMD at the Net level, in addition to the Region and Zone levels. 4.0 Definition of a "Grunged" Message -------------------------------------- What is a "grunged" message? This is a fuzzy question so it gets a fuzzy answer. A grunged message is a message which does not reasonably conform to FTSC-0001 and FTSC-0004 standards. It turns out that such strict conformance is not desired because so many messages fail to strictly conform to the standards (e.g., strict conformance to the standard requires two spaces between the year and time in the date/time stamp field of the message - it is fairly common to have only one space). Thus strict conformance would result in detecting messages otherwise generally considered acceptable. One of the classic symptoms of a grunged message is non-ASCII characters throughout the message. Fortunately, this condition is easy to detect in a message's control fields, especially the date field which is somewhat rigidly defined. Another form of grunging, related to a message becoming an unwanted duplicate, results in additional control fields being appended to the end of the message. Although not actually a grunged message, sometimes old messages will get scanned out again, resulting in duplicates which are too old for conventional dupe checking techniques to detect. GMD is capable of detecting these sorts of problem messages. 5.0 Operation -------------- 5.1 GMD -------- GMD scans incoming packets. It should be run after unpacking incoming bundles, perhaps with a program like PolyXArc, but before importing or tossing the mail. GMD tests each message for conformance with the criteria specified in the configuration file. When GMD detects a non-conforming message it can be set to do nothing or to perform any combination of the following: 1) Generate an "information" net mail message to the SysOp of the originating node, informing him of the problem. 2) Delete the non-conforming message from the incoming packet. 3) Make a "save" copy of the non-conforming message for a local "bad" area. GMD only operates on the incoming packets. The original incoming packets maintain their time/date, even if messages have been deleted from them. Any messages that GMD generates are put into new packets, also to be processed by your regular mail program. GMD does not work with your BBS's message base. Thus it does not care how your messages are actually stored on your system, or if they are stored at all (passthrough). For security reasons, some mail processors will not accept the messages which GMD generates since they appear to be coming from that very system itself. In this case GMD should be configured to generate it's messages as if it is coming from a Point off your system, rather than your system itself. This address is set by the System.xxx parameters in the Default Message Standards Section of the configuration file. Whatever you set the System.xxx address to, you will need to make sure that your mail processor will accept routed net mail (the "information" messages) and echo mail (the "save" messages) from this address. GMD puts its product code (A3) into the header of the packets as it inspects or generates them. GMD will not process a packet which has its own product code. Thus GMD will not process the same incoming packet twice, or any packet which it generated. GMD can be called with two optional command line parameters: -c Specifies a configuration file other than the default of GMD.Cfg in the current directory, or the directory where GMD.Exe resides. -t Specifies "test mode" operation. Used for testing GMD's setup. Inhibits the "i" and "d" commands. Only the "s" command is functional. This allows you to test your GMD setup without disturbing the mail flow or generating information messages. GMD should be configured to ignore incoming mail from other systems which you know are also running GMD. This helps to avoid sending out redundant "information" messages. When you set up GMD you should inform all of your echo mail connections that you are running it, and you should ask them to let you know if they are, too. You should then list these addresses in the Ignore_Packet_From parameter in the General Operation Section of the configuration file. Please see section 6.1 for more information. 5.2 GMD-Sort ------------- GMD generates a statistical database when it operates. The GMD-Sort program can be used to sort the database, prior to generating reports from it with the GMD-Rpt program. The database is sorted and replaced. This does not affect how GMD itself operates. It merely changes the order of the items in the report. To use GMD-Sort, call it with the name of the statistical database and optionally with one or more of the sort keys. The sort keys must be separated by at least one space. Each may be preceded by a "+" (default) or "-", which causes the sort to happen in ascending or descending order for that key. The default sort order, if you don't use any of the keys, is "+zone +net +node +point". The available sort keys are: zone save subject net block body node uninf gated point datetime origin info to seenby delete from path 5.3 GMD-Rpt ------------ GMD generates a statistical database when it operates. The GMD-Rpt program can be used to generate a report from it. To use GMD-Rpt, call it with the name of the statistical database. The report will be sent to the console. Redirection can be used to send the report wherever you desire. For example, to send it to a file called GMD.Rpt: GMD-Rpt GMD.Dat >GMD.Rpt 6.0 GMD Configuration File --------------------------- The configuration file contains all the parameters that control GMD. The GMD distribution file contains a sample configuration file, called Sample.Cfg. Blank lines and any part of a line following a ";" are considered comments, and are ignored by GMD. The configuration file is divided into four major sections: General General system information Sets Defines legal zones and character sets Default Defines the default settings for all echo areas Area Exceptions for groups of echo areas Each of these sections starts with appropriate keyword, immediately followed by a ":". Each section is made up of a series of parameters. These parameters can be in any order. A parameter consists of a key word, a "=", and the parameter's value(s). For example: Section_1: parameter_1 = 12 parameter_2 = 34, "abc", y parameter_3 = "John Doe" Any string which contains embedded spaces or other punctuation, must be quoted. If in doubt - quote it, as it won't hurt. Any filespec can include a path in addition to the file's name. Most parameters test for a condition. These parameters are followed by an "ids" command(s) which tells GMD what to do if the condition is not met. The "ids" commands are: i Inform the originator (via net mail) about the problem d Delete the bad message from the incoming packet s Save a copy of the bad message Any combination (including none if no action is desired) of the "ids" commands may be used, except that if the "d" command is used, then the "i" command will also be executed, even if you do not specify it. Some parameters only allow the "is" commands, and not the "d" command. If a message fails more than one condition, then the action that GMD takes is the "or" of the individual conditions' command settings. For example, if a message fails 3 condition tests, yet only one of these conditions specifies that a information message be sent, it will be sent. There are times when GMD is not able to send an information message, even though one is requested. This only occurs when GMD is unable to determine the originator's address. GMD gets this address from the origin line, if possible. GMD can also be configured to try the "MSGID:" hidden line. GMD permits include files anywhere a token (a string of characters which GMD recognizes as a unit) can appear. The syntax is @filespec (or "@filespec"). Include files can be nested. Included files can be handy for separate, but similar, configuration files. They can also be used to input lists of various sorts (echo areas, names, search strings). 6.1 General Operation Section ------------------------------ This section contains general information about your system and how GMD should operate. GMD logs to both the screen and to a file. You can specify the level of detail you want for each. GMD generates some summary statistics about the numbers and types of non-conforming messages. This information can be appended to the logs or even sent to you in a net mail message. The only disk directory that GMD needs to know about is the one which contains the incoming packets. This is usually your inbound mail area. GMD does not care what type of message base your system uses. Any messages that GMD itself generates will be put in the form of incoming packets. Thus your system treats them like any other incoming mail. This means that GMD also does not care what type of outbound area(s) your system uses. When GMD "saves" a copy of a non-conforming message, it does this by sending a copy of the message to a "bad" echo area which you can specify. This allows you to look at the messages later. GMD is capable of generating a database which contains statistical information about the number and types of errors encountered for each node. This database has two uses. It can be used to limit the number of information messages sent to a given node. Also it serves as input to the GMD-Sort and GMD-Rpt reporting programs. The database is cleared by deleting the file. A typical use would be to generate a weekly report, then delete the database. To prevent your GMD from sending out information messages which have already been sent out by another node running GMD, you can instruct GMD to "ignore" packets coming in from certain Nodes. In fact, GMD will continue to inspect these packets, but will not generate any information messages or save copies, unless a delete is also called for, in which case it processes the packet normally. This allows for the possibility of a message being grunged after it has been processed by a node running GMD. Please see section 5.1 for more information. We realize that this is not a foolproof scheme, although it should help in most cases. Future versions of GMD might use some sort of kludge line to implement the "ignore" logic. 6.2 Set Definitions Section ---------------------------- This section contains definitions of sets which are used in the Default and Area Sections. Sets are used to specify character sets and zone numbers, for example. You can name them whatever you want to and you can have as many as you need. Sets make it possible to specify any combination of characters or numbers which you wish to allow. For example: Just basic, printing, ASCII characters. Or you might want to allow some (the alphabetic), but not all, of the hi-bit characters, too. 6.3 Default Message Standards Section -------------------------------------- This section contains the default settings used for the echo areas. These set the parameters which determine whether a message conforms or not. These settings can be overridden for specific echoes by using the Area Section. Most of these parameters test conditions, hence use the "ids" commands. These commands determine what GMD does if a non-conforming message is found. Many of the parameters allow you to relax or disable GMD's conformance checking of various aspects of the message specifications. At present, there are so many non-conforming messages that we felt this was necessary until the majority of the responsible programs were fixed. Please see section 5.1 for a discussion about how to set the System.xxx parameters. GMD uses two template files to build the information messages that it sends. One template file is used for messages which are passed and another template file is used for messages which are deleted. There are a number of macros available which cause GMD to insert the appropriate information about the offending message. These are the macros which can be used in the template files (note the trailing "."): &To. The non-conforming message's "to" field &From. The non-conforming message's "from" field &Subject. The non-conforming message's "subject" field &Date. The non-conforming message's "date" field &Flags. The non-conforming message's flags &Why. The reason(s) that the message didn't conform &Area. The echo area that the non-conforming message was in &Body. The non-conforming message's body (up to the origin line) &Routing. The non-conforming message's control fields (from the origin line on) &SysOp. The name specified in System.SysOp Any other text in the template files is used in the information message "as is". The GMD distribution file contains two sample template files. Sample.Pas is an example of the Info_Message.Pass_File and Sample.Del is an example of the Info_Message.Delete_File. Some parameters require sets to represent lists of acceptable characters. Minimum character sets are OR'ed with the character sets that you use to prevent leaving out basic characters. The minimum character set is 32 (space) to 126 for the following parameters: To.Alphabet, From.Alphabet, Subject.Alphabet, Gated_Origin.Alphabet and Origin.Alphabet. In addition to 32 to 126, the minimum character set for the Body.Alphabet parameter also includes 1 (control A), 9 (tab), 10 (line feed) and 141 (soft carriage return). 6.4 Exception Areas Sections ----------------------------- These are optional sections which contain exceptions to the defaults listed in the Default Section. Any setting listed there can be overridden for specific echoes. An echo mail area can only be listed in one Exception Area Section. The only required parameter in an Area Section is "Echoes". It lists the echo mail areas for which the exceptions which follow it apply. Typical uses of Exception Areas include: A Node in more than one Network who has to cope with different message standards, a Moderator who wants to automatically send out a special information message to anyone who uses a non-ASCII character in the body of a message, or a Node or software developer who wants to do more stringent testing on certain test areas. 7.0 Runtime Considerations --------------------------- GMD returns an errorlevel of 0 if it executes normally. If it detects an error in the configuration file it will return an errorlevel other than 0. 7.1 DOS -------- Use the EXE version on 8086/8 CPUs. The 286 version runs a bit faster, but it can only be used on 80286, and above, CPUs. GMD requires about 350K of free memory to run. It is not overlaid, and does not use EMS or any other type of memory. GMD running on a 25 MHz 80486, under DesqView, takes about 5 minutes to process 9,500 messages. This is about one full day's mail for the Zone 1 Backbone as of September 1991. 7.2 OS/2 --------- Use the OS2 version of the programs. They are bound in the dual mode, thus work in either protected or real mode. 7.3 Other Operating Systems ---------------------------- We would be happy to work with anyone who is interested in porting GMD to other operating systems. Contact the author. 8.0 Licensing Restrictions --------------------------- There are some simple restrictions associated with using the programs. You use these programs at your own risk. The author does not warrant that the programs work as described herein or that they are even suitable for any particular purpose. This documentation is intended to describe how the programs work in the author's environment. The author does not claim that they will work the same way in your environment or, if they do work, that they will continue to work. Furthermore, it is up to you to decide if these programs are suitable for any particular purpose on your system. The author freely releases the executable versions of these programs to the public domain. The author desires no fees for the use of these programs and does not support these programs. These programs compile and link using Borland C++ and Microsoft C 6.0 with no errors. The source codes for GMD and related programs are not generally available. However, if you have a legitimate need, and are willing to sign a non-disclosure agreement, contact the author. GMD has been assigned FTSC product code "A3". 9.0 Credits ------------ David Troendle, 1:396/5, is the author of GMD and related programs. As time permits, he is willing to answer questions about techniques used and how these program generally work. John Souvestre, 1:396/1, is responsible for the testing and documenting of GMD and related programs. He also serves as the interface to the author, when David is busy otherwise. We would like to thank the many people who helped in one way or another with GMD's development. Thanks to the many participants of the ZEC echo (prior to the GMD echo's formation) who supplied critical review and suggestions about what GMD should and should not do. Thanks to the following people who supplied helpful coding ideas or actual code: Chris Irwin 1:18/68 Jeffrey Nonken 1:273/715 George Peace 1:13/13 Thanks to the alpha test team: Tony Davis 1:147/100 John Souvestre 1:396/1 Thanks to the beta test team: Steve Ahola 1:322/1 Bill Bolton 3:711/403 Rod Bowman 1:10/8 Tim Carter 1:142/222 Fabian Gordon 1:107/323 Miles Hoover 1:109/25 Felix Kasza 2:310/11 Dean Lachan 1:124/4115 Paul Marwick 3:640/820 David Nugent 3:632/348 Mike Ratledge 1:372/666 Matt Whelan 3:54/99 Ken Wilson 1:12/12 Thanks to the GMD Echo Moderator Steve Shapiro, 1:382/35. This is the best place to get information about GMD or help in using it. 10.0 History of Changes ------------------------ Version Date Changes, Modifications, Additions, etc. ------- -------- --------------------------------------- 1.00 12/29/90 Original release 1.01 2/16/91 Maintenance release, a few small fixes 2.00 2/28/91 Added features mainly for Hub usage - Uses echotoss log - Uses and updates lastread pointer - Optional passthrough area processing - Configurable log detail level - Returns an errorlevel 3.00 9/01/91 Complete redesign and rewrite! - Processes incoming packets directly - Many new parameters - Parameters now in configuration file - Ability to send information messages - OS/2 version - Statistical database and reporting - "Delete" command disabled for now - End -